Distributed Data Processing Research Articles

Sales And Production Cost Applications At Pt. Semarang Delta

In processing sales and production data at PT. Delta, is an important activity, where distributed data processing still uses Microsoft Excel and Microsoft Word. Therefore, to make it easier for employees to enter data and sell goods, the author tries to process the data using a computerized system that is more effective and efficient.Data collection and analysis were carried out in an effort to solve this problem. The research method used in making this system, from data collection, problem analysis, system design to system creation, is the observation method, and directly handling sales at PT. Delta. This application system was created using the PHP programming language and the database was processed using Adobe Dreamweaver, Appserv, and MySQL.By using this system, the distribution of sales and production data processing at PT. Delta will be easier and more detailed, making it very easy for employees.

Evaluation of distributed data processing frameworks in hybrid clouds

Distributed data processing frameworks (e.g., Hadoop, Spark, and Flink) are widely used to distribute data among computing nodes of a cloud. Recently, there have been increasing efforts aimed at evaluating the performance of distributed data processing frameworks hosted in private and public clouds. However, there is a paucity of research on evaluating the performance of these frameworks hosted in a hybrid cloud, which is an emerging cloud model that integrates private and public clouds to use the best of both worlds. Therefore, in this paper, we evaluate the performance of Hadoop, Spark, and Flink in a hybrid cloud in terms of execution time, resource utilization, horizontal scalability, vertical scalability, and cost. For this study, our hybrid cloud consists of OpenStack (private cloud) and MS Azure (public cloud). We use both batch and iterative workloads for the evaluation. Our results show that in a hybrid cloud (i) the execution time increases as more nodes are borrowed by the private cloud from the public cloud, (ii) Flink outperforms Spark, which in turn outperforms Hadoop in terms of execution time, (iii) Hadoop transfers the largest amount of data among the nodes during the workload execution while Spark transfers the least amount of data, (iv) all three frameworks horizontally scale better as compared to vertical scaling, and (v) Spark is found to be least expensive in terms of $ cost for data processing while Hadoop is found the most expensive.

A MapReduce-Based Approach for Fast Connected Components Detection from Large-Scale Networks.

Owing to increasing size of the real-world networks, their processing using classical techniques has become infeasible. The amount of storage and central processing unit time required for processing large networks is far beyond the capabilities of a high-end computing machine. Moreover, real-world network data are generally distributed in nature because they are collected and stored on distributed platforms. This has popularized the use of the MapReduce, a distributed data processing framework, for analyzing real-world network data. Existing MapReduce-based methods for connected components detection mainly struggle to minimize the number of MapReduce rounds and the amount of data generated and forwarded to the subsequent rounds. This article presents an efficient MapReduce-based approach for finding connected components, which does not forward the complete set of connected components to the subsequent rounds; instead, it writes them to the Hadoop Distributed File System as soon as they are found to reduce the amount of data forwarded to the subsequent rounds. It also presents an application of the proposed method in contact tracing. The proposed method is evaluated on several network data sets and compared with two state-of-the-art methods. The empirical results reveal that the proposed method performs significantly better and is scalable to find connected components in large-scale networks.

Scalable Data Partitioning Techniques for Distributed Data Processing in Cloud Environments: A Review

Scalable Data Partitioning Techniques for Distributed Data Processing in Cloud Environments: A Review

RelJoin: Relative-cost-based selection of distributed join methods for query plan optimization

Selecting appropriate distributed join methods for logical join operations in a query plan is crucial for the performance of data-intensive scalable computing (DISC). Different network communication patterns in the data exchange phase generate varying network communication workloads and significantly affect the distributed join performance. However, most cost-based query optimizers focus on the local computing cost and do not precisely model the network communication cost. We propose a cost model for various distributed join methods to optimize join queries in DISC platforms. Our method precisely measures the network and local computing workloads in different execution phases, using information on the size and cardinality statistics of datasets and cluster join parallelism. Our cost model reveals the importance of the relative size of the joining datasets. We implement an efficient distributed join selection strategy, known as RelJoin in SparkSQL, which is an industry-prevalent distributed data processing framework. RelJoin uses runtime adaptive statistics for accurate cost estimation and selects optimal distributed join methods for logical joins to optimize the physical query plan. The evaluation results on the TPC-DS benchmark show that RelJoin performs best in 62 of the 97 queries and can reduce the average query time by 21% compared with other strategies.1

Improving Data Collection in Pregnancy Safety Studies: Towards Standardisation of Data Elements in Pregnancy Reports from Public and Private Partners, A Contribution from the ConcePTION Project.

The ConcePTION project aims to improve the way medication use during pregnancy is studied. This includes exploring the possibility of developing a distributed data processing and analysis infrastructure using a common data model that could form a foundational platform for future surveillance and research. A prerequisite would be that data from various data access providers (DAPs) can be harmonised according to an agreed set of standard rules concerning the structure and content of the data. To do so, a reference framework of core data elements (CDEs) recommended for primary data studies on drug safety during pregnancy was previously developed. The aim of this study was to assess the ability of several public and private DAPs using different primary data sources focusing on multiple sclerosis, as a pilot, to map their respective data variables and definitions with the CDE recommendations framework. Four pregnancy registries (Gilenya, Novartis; Aubagio, Sanofi; the Organization of Teratology Information Specialists [OTIS]; Aubagio, Sanofi; the Dutch Pregnancy Drug Register, Lareb), two enhanced pharmacovigilance programmes (Gilenya PRIM, Novartis; MAPLE-MS, Merck Healthcare KGaA) and four Teratology Information Services (UK TIS, Jerusalem TIS, Zerifin TIS, Swiss TIS) participated in the study. The ConcePTION primary data source CDE includes 51 items covering administrative functions, the description of pregnancy, maternal medical history, maternal illnesses arising in pregnancy, delivery details, and pregnancy and infant outcomes. For each variable in the CDE, the DAPs identified whether their variables were: identical to the one mentioned in the CDE; derived; similar but with a divergent definition; or not available. The majority of the DAP data variables were either directly taken (85%, n = 305/357, range 73-94% between DAPs) or derived by combining different variables (12%, n = 42/357, range 0-24% between DAPs) to conform to the CDE variables and definitions. For very few of the DAP variables, alignment with the CDE items was not possible, either because of divergent definitions (1%, n = 3/357, range 0-2% between DAPs) or because the variables were not available (2%, n = 7/357, range 0-4% between DAPs). Data access providers participating in this study presented a very high proportion of variables matching the CDE items, indicating that alignment of definitions and harmonisation of data analysis by different stakeholders to accelerate and strengthen pregnancy pharmacovigilance safety data analyses could be feasible.

Towards an Energy Complexity Model for Distributed Data Processing Algorithms

Towards an Energy Complexity Model for Distributed Data Processing Algorithms

MR-AT: Map Reduce based Apriori Technique for Sequential Pattern Mining using Big Data in Hadoop

One of the most well-known and widely implemented data mining methods is Apriori algorithm which is responsible for mining frequent item sets. The effectiveness of the Apriori algorithm has been improved by a number of algorithms that have been introduced on both parallel and distributed platforms in recent years. They are distinct from one another on account of the method of load balancing, memory system, method of data degradation, and data layout that was utilised in their implementation. The majority of the issues that arise with distributed frameworks are associated with the operating costs of handling distributed systems and the absence of high-level parallel programming languages. In addition, when using grid computing, there is constantly a possibility that a node will fail, which will result in the task being re-executed multiple times. The MapReduce approach that was developed by Google can be used to solve these kinds of issues. MapReduce is a programming model that is applied to large-scale distributed processing of data on large clusters of commodity computers. It is effective, scalable, and easy to use. MapReduce is also utilised in cloud computing. This research paper presents an enhanced version of the Apriori algorithm, which is referred to as Improved Parallel and Distributed Apriori (IPDA). It is based on the scalable environment referred as Hadoop MapReduce, which was used to analyse Big Data. Through the generation of split-frequent data regionally and the early elimination of unusual data, the proposed work has its primary objective to reduce the enormous demands placed on available resources as well as the reduction of the overhead communication that occurs whenever frequent data are retrieved. The paper presents the results of tests, which demonstrate that the IPDA performs better than traditional apriori and parallel and distributed apriori in terms of the amount of time required, the number of rules created, and the various minimum support values.

WITHDRAWN: Efficient fragmentation and allocation on clustering in distributed environment (FACE)

Distributed data processing systems require a reliable storage structure that ensures consistent, high-scale, and scalable data access for all users. Data fragmentation and subsequent allocation are important aspects of a flexible distributed database system. Existing techniques may result in significant data loss while performing data allocation and fragmentation. To overcome these issues a novel fragmentation and Allocation on Clustering in Distributed Environment (FACE) is proposed for reliable storage with scalable data access for all users. Initially, the user query is given to distributed database system (DDBS), then the query is fed to the query parser. The query parser parses and converts a textual query into a query object. After parsing the query data, it is passed to the query optimizer, which optimizes the query to access the data more efficiently. In the next step, the query is fragmented using dual (horizontal and vertical) fuzzy C means clustering. In Dual fuzzy C means clustering, the vertical fragmentation focuses on clustering attribute values that are frequently accessed together, and horizontal fragmentation focuses on grouping records that meet data selection criteria. After fragmentation, the query is allocated using sequential rule mining, in which queries occur in sequential order. The allocated query is optimized using the Efficient Fragment Security (EFS) algorithm to normalize the parameters to achieve better results. The performance metrics of the proposed techniques are calculated based on Final Prediction Error (FPE), and Execution time. The proposed FACE achieves a lower execution time of the 1.56 s. The result shows that the proposed FACE decreases the overall execution time lower than the 4.70 s, 4.46 s, 3.3 Sec, and 3.14 s in MAPA, DMA, BBE, and RATS-HM respectively.

Distributed Data Processing: A Scalable and Cost-Effective Approach

Distributed Data Processing: A Scalable and Cost-Effective Approach

Joint Optimization of Memory Sharing and Communication Distance for Virtual Machine Instantiation in Cloudlet Networks

Cloudlet networks are an emerging distributed data processing paradigm, which contain multiple cloudlets deployed beside base stations to serve local user devices (UDs). Each cloudlet is a small data center with limited memory, in which multiple virtual machines (VMs) can be instantiated. Each VM runs a UD’s application components and provides dedicated services for that UD. The number of VMs that serve UDs with low latency is limited by a lack of sufficient memory of cloudlets. Memory deduplication technology is expected to solve this problem by sharing memory pages between VMs. However, maximizing page sharing means that more VMs that can share the same memory pages should be instantiated on the same cloudlet, which prevents the communication distance between UDs and their VMs from minimizing, as each VM cannot be instantiated in the cloudlet with the shortest communication distance from its UD. In this paper, we study the problem of VM instantiation with the joint optimization of memory sharing and communication distance in cloudlet networks. First, we formulate this problem as a bi-objective optimization model. Then, we propose an iterative heuristic algorithm based on the ε-constraint method, which decomposes original problems into several single-objective optimization subproblems and iteratively obtains the subproblems’ optimal solutions. Finally, the proposed algorithm is evaluated through a large number of experiments on the Google cluster workload tracking dataset and the Shanghai Telecom base station dataset. Experimental results show that the proposed algorithm outperforms other benchmark algorithms. Overall, the memory sharing between VMs increased by 3.6%, the average communication distance between VMs and UDs was reduced by 22.7%, and the running time decreased by approximately 29.7% compared to the weighted sum method.

Communication-Efficient Federated Double Distillation in IoV

6G will be the next horizon from connected people and things to intelligence-of-everything. Machine learning (ML) combines artificial intelligence with vehicular edge computing (VEC), further releasing the potential of data. Federated learning (FL) is used to manage the collaborative training of vehicular clients and the road side unit (RSU), which can efficiently complete the distributed data processing. However, the vehicular clients upload a large number of model parameters to occupy regular communication resources, and the stability of the link cannot be guaranteed due to the mobility of the vehicles, which may lead to expensive communication overhead in VEC. Thus, this paper proposes a communication-efficient federated double distillation (FedDD) framework, which comprehensively considers the three-dimensional attributes of the vehicles and dynamically selects the cluster-heads (CHs) to improve the transmission efficiency. Then, knowledge distillation is further integrated into the federated learning to accomplish multiple rounds of parameter distillation, thereby significantly reducing the communication overhead. Experimental results show that compared to traditional FedAvg, FedDD can reduce communication overhead by three orders of magnitude. The communication overhead of FedDD is reduced by 82% compared to FTTQ, which improves the communication efficiency of FL while sacrificing a small amount of accuracy.

Construction and Application of Trajectory Data Analysis Model Based on Big Data and Stochastic Gradient Descent Algorithm

This paper studies the model construction of computing and storage resource management system framework based on Hadoop and the implementation of trajectory data analysis function under big data. Relying on the cloud platform infrastructure, in order to support the rapid data growth and massive data processing needs, it provides a mixed storage and analysis platform for structured and unstructured data, and uses big data technology to build a highly scalable and distributed data processing framework. The distributed computation, overall frame model of the memory system, and function module have been built with the aim of constructing the system in consideration. Second, by using Hadoop to preprocess the original data and concentrating on the data hierarchical design model and key technology analysis of big data systems, the design model, functional modules, technological solutions, and SGD algorithm are suggested, along with the detailed implementation procedure. Lastly, by merging the data of running vehicles, the system accomplishes the data analysis of vehicle trajectory, empty and load cars, and load and unload people.

Method of planning data processing tasks in distributed systems with limited information about available resources

In today's digital landscape, distributed data processing systems (DDPs) are becoming increasingly critical to efficiently process, analyze, and manage large volumes of data. These systems are often used in commercial, scientific and social domains to process complex data in real-time or batch mode. One of the key components of such systems is task scheduling, which is an extremely complex process, particularly when information about resource requirements is not complete or accurate. The subject of research are algorithms, methods and approaches used for scheduling tasks between nodes in distributed systems. The purpose of the study is to create an optimized method of task planning in the RSOD with limited availability of information about available resources. The task of the research: to analyze the limitations of modern methods for scheduling tasks in distributed data processing systems (DDS); optimize the method of scheduling tasks based on metadata between RSOD nodes, based on the methodology of searching for nearest neighbors using the method of localized hashing and the algebra of finite predicates; develop the architecture of the software solution and its implementation based on the optimized method; test the algorithm on the example of a video decoding task. The following methods were used: statistical algorithms and techniques such as classification and cluster analysis were used to predict resource requirements, and visualization techniques assisted in the analysis and interpretation of results. As a result of the work: the limitations of modern methods for the distribution of tasks in distributed data processing systems (DDPs) were analyzed; an optimized method of task planning based on metadata in RSOD was created, based on the methodology of searching for nearest neighbors using the method of localized hashing and the algebra of finite predicates; the processes in the modified nearest neighbor search algorithm are detailed; the architecture of the software solution was developed, which integrates an optimized method of task planning based on metadata and resource allocation; validation of the software solution was carried out with the help of a practical scenario – the use of the created algorithm in the planning task for decoding video information. The conclusions of this study confirmed that the proposed method, based on the methodology of localized hashing and the use of finite redicate algebra, is effective even with insufficient or limited information about resource needs. This highlights the possibility of using dynamic scheduling strategies to adapt to changing load conditions and resource availability.

FIDEL: Fog integrated federated learning framework to train neural networks

AbstractTechnological advancement in the digital era has continued to produce voluminous amounts of data through various devices. Even though data is produced distributively, it needs to be accumulated centrally for processing, analysis, and knowledge extraction that faces several challenges such as bandwidth, latency, congestion, privacy, and security. Fog computing paradigm addresses some of these issues, and can be used as a distributed data processing unit. Federated learning trains a shared model over distributed nodes. However, a fog node can not process continuously growing data due to computational limitations. In this paper, we propose FIDEL: a fog integrated federated learning framework for neural network training using resource‐constrained devices. The federation of resource‐constrained Internet of Things (IoT) devices creates a shared global model trained on local data, which is generalized on the unseen dataset for prediction/inferences. We have also designed an online training scheme to process continuous data with limited compute resources. The FIDEL supports both synchronous and asynchronous federate learning that empowers resource‐constrained devices to train machine learning models. To test the learning capabilities of the FIDEL, we have trained three neural networks (i) Shallow network; (ii) Deep Network; (iii) Convolutional Neural Network (CNN) models for human position detection in industrial IoT setup on rapidly changing datasets. The experimental results show that the framework can learn input–output relationships with significantly high accuracy. The overall system efficiency of the framework is reasonable in terms of latency and memory usage for resource‐constrained devices.

BIG DATA PROCESSING WITH APACHE SPARK

With the exponential growth of information, it is no surprise that we are in a period of history as the Information Age. The rapid growth of data has presented challenges regarding storage and processing technology. This article refers to Apache Spark, an ecosystem that provides many integrated technologies in Big Data processing, including machine learning libraries and data storage platforms. Apache Spark provides distributed data processing for open source applications, loading data in-memory and making operations for analyzing data of any size, with efficient support for popular programming languages like Java, Scala, R, and Python. The article aims to compare the superior computing power of Saprk compared to Hadoop and how to connect Spark with today's popular data processing tools such as the R language.

Networking of Sensors With Fixed Transmission Ranges: Distributed Data Processing Over Graphs

In this letter, we address the data gathering problem in low-power wireless sensor networks (WSNs) with sensors of fixed transmission ranges. We model the NP-hard problem as an integer linear program that can be applied to networks of small sizes. Further, we propose a novel graphical model that apprehends the features of such networks, and design a polynomial-time approximation algorithm using the proposed graphical model for networks of large sizes. The performance is evaluated under data compaction strategies based on compressive sensing. Simulation results demonstrate the superior performance of the proposed method over its peer protocols.

Data Compression-Aware Performance Analysis of Dask and Spark for Earth Observation Data Processing

High-performance computing is a good choice for handling Big Earth Observation data, allowing the processing of the data in a distributed and performance-efficient way using in-memory computing frameworks. The data compression technique reduces the amount of storage and network transfer time and improves processing performance. The article aims to investigate the effectiveness of widely used distributed data processing frameworks in conjunction with lossless data compression techniques, to find the optimal compression method and processing framework for specific earth observation workflows. Normalized Difference Vegetation Index has been evaluated for the territory of Armenia, obtaining data from the Sentinel satellite and considering the supported compression methods to compare the performance of in-memory Dask and Spark frameworks. Experiments show that the Zstandard compression method and the Dask framework are the best choices for such workflows.

ReactiveFnJ: A choreographed model for Fork-Join Workflow in Serverless Computing

Function-as-a-Service (FaaS) is an event-based reactive programming model where functions run in ephemeral stateless containers for short duration. For building complex serverless applications, function composition is crucial to coordinate and synchronize the workflow of an application. Some serverless orchestration systems exist, but they are in their primitive state and do not provide inherent support for non-trivial workflows like, Fork-Join. To address this gap, we propose a fully serverless and scalable design model ReactiveFnJ for Fork-Join workflow. The intent of this work is to illustrate a design which is completely choreographed, reactive, asynchronous, and represents a dynamic composition model for serverless applications based on Fork-Join workflow. Our design uses two innovative patterns, namely, Relay Composition and Master-Worker Composition to solve execution time-out challenges. As a Proof-of-Concept (PoC), the prototypical implementation of Split-Sort-Merge use case, based on Fork-Join workflow is discussed and evaluated. The ReactiveFnJ handles embarrassingly parallel computations, and its design does not depend on any external orchestration services, messaging services, and queue services. ReactiveFnJ facilitates in designing fully automated pipelines for distributed data processing systems, satisfying the Serverless Trilemma in true essence. A file of any size can be processed using our effective and extensible design without facing execution time-out challenges. The proposed model is generic and can be applied to a wide range of serverless applications that are based on the Fork-Join workflow pattern. It fosters the choreographed serverless composition for complex workflows. The proposed design model is useful for software engineers and developers in industry and commercial organizations, total solution vendors and academic researchers.

Hand Gesture Interface for Robot Path Definition in Collaborative Applications: Implementation and Comparative Study.

The article explores the possibilities of using hand gestures as a control interface for robotic systems in a collaborative workspace. The development of hand gesture control interfaces has become increasingly important in everyday life as well as professional contexts such as manufacturing processes. We present a system designed to facilitate collaboration between humans and robots in manufacturing processes that require frequent revisions of the robot path and that allows direct definition of the waypoints, which differentiates our system from the existing ones. We introduce a novel and intuitive approach to human-robot cooperation through the use of simple gestures. As part of a robotic workspace, a proposed interface was developed and implemented utilising three RGB-D sensors for monitoring the operator's hand movements within the workspace. The system employs distributed data processing through multiple Jetson Nano units, with each unit processing data from a single camera. MediaPipe solution is utilised to localise the hand landmarks in the RGB image, enabling gesture recognition. We compare the conventional methods of defining robot trajectories with their developed gesture-based system through an experiment with 20 volunteers. The experiment involved verification of the system under realistic conditions in a real workspace closely resembling the intended industrial application. Data collected during the experiment included both objective and subjective parameters. The results indicate that the gesture-based interface enables users to define a given path objectively faster than conventional methods. We critically analyse the features and limitations of the developed system and suggest directions for future research. Overall, the experimental results indicate the usefulness of the developed system as it can speed up the definition of the robot's path.